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ABSTRACT 



Noting that having research evidence to recommend or refute 
specific policy choices is especially relevant in this era of increased 
demand for accountability in human services and government, this fact sheet 
details 10 criteria with which to evaluate the worth of research reports. The 
10 criteria or considerations are: (1) consider the source; (2) media are 

also sources to be evaluated; (3) has the research been published, and where? 
(4) research results are really about the topic "as measured," not as one may 
think it is; (5) different types of research have different strengths,* (6) 
sampling is more important that sample size,* (7) statistical significance 
explained; (8) research findings are about groups; (9) all research is not 
created equal; and (10) any one study is not the whole story. Most of these 
steps highlight portions of an "advocate's checklist," listing pertinent 
questions to ask about a given report. (HTH) 
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Understanding Research: 

TopTenTips for Advocates and Policymakers 

BY STEPHANIE A. SCHAEFER, PH D. 



R esearch crosses the desks of advocates and policy- 
makers on a regular basis. You receive a new report 
from a think tank or government agency, or you read 
a newspaper article describing the results from a new study. 

Research is an important tool because it allows us to assess 
the effectiveness of the wide array of policies and programs 
affecting the lives of children and families. Having research 
evidence to recommend or refute specific policy choices is 
especially relevant in this era of increased demand for account- 
ability in human services and government. 

But how can you tell if a given research study is one you can 
trust? Below are several tips that can help you to evaluate 
critically the research you encounter. 

1. Consider the source. 

It is important to evaluate the credibility of the individual(s) and 
the organization that produced the research. Research pro- 
duced by respected researchers and institutions is more likely to 
be trustworthy. Also, research produced or funded by groups 
with a strong political or commercial agenda (e.g., partisan 
groups, or the company which manufactured the product being 
studied) is less trustworthy, since these groups have a vested 
interest in the study’s findings supporting their viewpoint. 



ADVOCATE’S CHECKLIST 



i 



What do you know (or what can you find out) about 
the person and the organization diat did the research? 
What are the author’s research qualifications? 

What is the author’s reputation as a researcher? 



■ Is the researcher from a reputable organization, 
J university or research institute? 



Does the person or the organization have a political 
agenda they consistendy promote? 



■y 



2. Media is also a source to be evaluated . 

If you are learning about research through the media, keep in 
mind that the media coverage may not fully or accurately sum- 
marize the original research. Because research can be technical 
and complex, and because media coverage often seeks to be 
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attention-grabbing and succinct, media reporting of research 
sometimes oversimplifies the research, leading to misinterpre- 
tation. Don’t assume that media’s report of the research is 
necessarily what the actual study says, particularly if the media 
coverage is very brief or provocative. Do follow up by trying to 
get a copy of the original research article, or by getting more 
information from additional sources. 



ADVOCATE’S CHECKLIST 



If you learned about the research through the media: 

■ Was the media coverage very brief? 

If so, there may be more to the story than was 
addressed in the limited coverage. 

■ Was the reporting on it provocative? 

If so, you’ll want to determine if the research finding 
itself has controversial implications, or if the reporting 
of it played up that angle. 

To get more information on research that’s getting 
media coverage, try the following additional sources: 

■ Media coverage from additional sources, such as other 
newspapers, many of which are available on the Internet, 
may provide another perspective. 

■ The web site of the researcher’s organization, the 
organization sponsoring a conference at which the 
research was presented, or the journal or publication 
in which the research was published may have a press 
release or the full research paper available. 



3. Has the research been published, and where? 

Research published in peer-reviewed research journals 1 is more 
trustworthy because it has been scrutinized by other 
researchers before being published. For example, the Journal 
of the American Medical Association , a peer-reviewed research 



1 Research journals use a peer-review process, in which a research article submitted for pub- 
lication is given to several other researchers knowledgeable in the topic for critical review. 
These peer reviewers provide independent assessments of the research, and can recom- 
mend revisions, or that the article not lie published. You can tell if a journal is peer reviewed 
by looking at the information for authors submitting articles, which is generally included in 
every journal issue and on a journal’s web site. 

3 






journal, is considered a highly reputable source. Unpublished 
research, or research published in publications that don’t 
critically evaluate it, has not gone through such scrutiny, so 
you should put less trust in this research. For example, 
research presented at a conference generally has not yet been 
published, and thus should be viewed as preliminary until it 
goes through the full publication review process. This is also 
true for research that is “unpublished” in the research sense 
but nonetheless has been reported in the media. However, 
even good research starts out as unpublished work and is 
published later, so the fact that a study is unpublished does not 
mean that it is poor quality. Research published by credible 
research institutions, such as the Urban Institute, is acceptable; 
look to the reputation of the research institution as a guide to 
the trustworthiness of the research. 



ADVOCATE’S CHECKLIST 



■ Has the research been published? 
j If so: 

■ Does die publication use a peer-review process? 

■ How reputable is the journal in which it was published? 
(You can ask a researcher who works in diat field of study 
how well-respected the journal is.) 



4. Research results are really about the topic 
as. measured, not as iv e may think of it. 

In any research study, the topic studied is measured in some 
specific way. Knowing how the topic was measured helps you to 
understand what the research was really about. 

For example, a researcher may study child aggression. This 
topic 2 could mean a lot of different things to different people 
(calling someone names, or physically attacking someone, for 
example). Since a topic such as aggression can be so broadly 
defined, researchers always come up with a more specific, 
precise definition 3 of the topic they are studying. The defini- 
tion of aggression in a study could be the number of times 
the child displayed five specific behaviors (shouting, hitting, 
kicking, biting, pushing), as observed by researchers or as 
reported by the child’s teacher. 

When the results from a study are reported, the results are 
really about the precise definition (display of specific behav- 
iors observed by the teacher), rather than the larger topic 
(aggression). In reading research, you want to assess whether 
the way the researchers defined and measured their topic 
makes common sense. Much of the time, the specific defini- 
tion does make common sense and seems reasonable 
(aggression = hitting people), but on occasion, a study 
defines a term in an unusual way (aggression = name-calling). 
In the latter case, it is important to be aware of the definition, 
because the study may report its findings as being about the 
> Voider topic. 



Also, different studies may use different definitions for the 
same topic. It is important to pay attention to these definitions 
when you are comparing the results from different studies. 



A D VOC A T E’S C H E C K L I S T 



■ Flow was the research topic(s) defined 
I and measured in this study? 

■ Does the precise definition used make common sense? 

■ Did this study use a similar or different definition than 
other studies have for this topic? 



5 . Different types of research have 
different strengths . 

Another indicator of the quality of a research study, and the 
claims that can be made based on it, is the study’s research 
design. The research design is the way the study is structured 
to answer a question. There are two broader categories of 
research: quantitative research, and qualitative research. 
Quantitative research uses numbers, and analyzes and reports 
data in numeric form. Qualitative research typically reports 
results through story-like descriptions rather than numbers. 

Experimental design studies, a type of quantitative study, offer 
the strongest evidence about the impact of a program. In an 
experimental design, researchers randomly assign individuals 
from the same population to two groups, a treatment group 
and a control group, and then compare the two groups on 
some outcome. Experimental studies, known as the ‘gold stan- 
dard’ of research methodology, produce the strongest evidence 
that a program produced an effect. Experimental studies, 
sometimes called control group studies or experiments, are the 
only type of study that can show a causal relationship. 

Although experimental studies can provide the strongest 
evidence, there are limitations to the situations in which this 
research design can be used. Experiments are very expensive to 
conduct. Also, in the world of social policy, it is often impractical 
or unethical to assign children to different research treatment 
groups (children growing up with one versus two parents in the 
home, for example) to attain the control needed for an experi- 
mental study. Another limitation of experiments is that the 
results obtained under the carefully controlled research situ- 
ation may not occur in the same way when replicated out in the 
community. This issue is called generalizability . 

Quasi-experimental and survey studies are another type of 
quantitative research design that are useful for measuring the 
effects of different programs on children. Quasi-experimental 
studies do not use random assignment to create the groups 
being studied. Instead, they find comparable groups in which to 

2 The research topic being studied is ealled a construct. 

3 This precise definition is ealled an operational definition. 





study the effects of different programs. These studies can find 
associations between a program and children’s outcomes, but 
they cannot be used to establish a causal relationship. For exam- 
ple, a quasi-experimental study may find that children who par- 
ticipated in an enrichment program had better social skills than 
those that did not, but it cannot prove that the program caused 
the increase in social skills (perhaps the children who partici- 
pated in the program had better social skills to begin with). 
Quasi-experimental studies are especially useful for studying 
complex systems as they exist naturally in the community. They 
are the best approach for large-scale studies which study larger 
numbers of people and which study more topics. 

Although advocates less often use qualitative research, it is 
another useful approach. Qualitative research, which typically 
reports data in non-numeric form such as categories or descrip- 
tions, can be an important source of information. Qualitative 
studies often provide descriptive, story-like accounts of people’s 
experiences in a program or in a community. Qualitative 
research is particularly well-suited to finding out new things 
you didn’t know to look for and ask about in a survey. 



general guideline for an acceptable response rate would be 
50%, and a very good response rate would be 80% or higher. 



ADVOCATE’S CHECKLIST 



■ What is the sample size used in the study? 

■ Flow was the sample selected? 

■ What was the response rate for the study? 



7 . Statistical significance explained. 

One of the things advocates value most about research is getting 
“hard data” - numbers - about the effects of a policy on children. 
A study reports a statistically significant difference between those 
who received a program and those who didn’t. But what does 
statistical significance mean, and what can we conclude from it? 

A statistically significant result is one that is unlikely to be due 
to chance. Researchers use statistics to test whether the results 
they found are likely to be due to the effect of the program 
being studied, and not to other unrelated factors. 



6. Sampling is more important than sample size. 

As many advocates know, the study’s sample size" 4 is important. 
The minimum sample size needed in quantitative research 
depends on how big the effects being studied are, so there is no 
rule, but a general guideline for a minimum sample size might 
be 30 to 50 people. The larger the sample, the smaller the dif- 
ference needed between groups to attain statistical significance. 



But even more important than sample size is the way the sample 
was collected. Quantitative research is based on the assumption 
that the findings for a sample of people can be generalized to the 
larger population. Researchers collect information on a sample 
of people in order to determine the effects of a program for the 
full population. For example, a study will select a sample of 100 
children in afterschool programs, and this sample is intended to 
represent the population of all children in similar programs. 
Researchers use careful procedures to select dieir samples. One 
appropriate procedure, the most commonly used, is random 
selection, but there are other appropriate sampling procedures 
as well. If the sampling procedures aren’t done well, then we 
cannot assume that the findings for the sample generalize to the 
population, and the study’s findings would not be valid. 



One important aspect of good sampling is the response rate. 5 
If a study has a low response rate, then this means that a portion 
of the carefully selected sample was not studied. It is possible 
that the people who did not respond are different in some 
systematic way from the people who did respond. For example, 
in a written survey, people who do not answer the questions 
might have lower literacy skills than the people who did 
answer it. The response rate is very important for this reason. 
® ile there are no hard and fast rules on response rates, a 



Let’s take a hypothetical example: a study found that children 
who received preventive health services had significantly 
higher rates of school attendance than children who did not 
have access to these services. Specifically, 75% of children who 
had health care had good school attendance, but only 50% of 
children without preventive health care had good attendance. 
That this finding was statistically significant means that it is 
highly unlikely that the difference found between the two 
groups was due to chance; therefore, it is likely that the dif- 
ference between the groups was really due to the difference in 
access to health care. 

Statistical significance is different than the substantive signifi- 
cance, or meaningfulness, of a finding. A result may be statis- 
tically significant but unimportant (sample size is crucial here, 
because a very small difference will be statistically significant 
if group sizes are large). Conversely, a result may not be sta- 
tistically significant, perhaps because the sample size was too 
small, but it may be meaningful nonetheless because it suggests 
an important change in an outcome. 



8. Research findings are about groups. 

Research results are usually based on comparisons between 
groups of people. For example, a study may find that children 
in program X have higher reading scores than children in 
program Y. That research findings are based on groups of 
people makes them particularly relevant for policy decisions, 
since policies affect groups of people, but less relevant for 
individual case decisions. 

4 The sample size is the number of people included in the study. 

5 The response rate is the proportion of people who were selected to be in the study 
compared to those that actually participated. 






In addition to looking at the difference between two groups, 
it is also worthwhile to look at the absolute levels of perform- 
ance in each group when deciding what this research tells you. 
Let’s say that 85% of program X children and 70% of program 

Y children read at a fifth-grade level. Let’s also assume that 
this is a statistically significant difference, and a meaningful 
difference as well. While it is true that program X children 
did better, it is also important to note that most of the program 

Y children also are reading at this level (it would be a different 
story if only 10% of program Y children were reading at the 
fifth-grade level). Knowing how big a difference there is 
between the two groups studied, and what the absolute levels of 
performance are for each group, taken together can help you 
make more informed policy decisions. 



ADVOCATE’S CHECKLIST 



■ For a finding comparing two groups, what were the 
| absolute levels of performance for each group? 



9. All research is not created equal. 

When comparing the results from different studies with con- 
flicting findings, higher-quality studies should be given more 
weight (you can use the tips provided in this fact sheet as a 
guide to determining the quality of a study). Better studies can 
refute poorer studies; there is not a one-to-one comparison. 

1 0. Any one study is not the whole story. 

Although we usually come across research one study at a time, 
from the news or a new report, research is most valuable when 
many specific studies are taken together to tell the whole story 
of what we know on a given topic. Research, as a tool for scien- 
tific discovery, is designed to work this way. Science is about the 
aggregation of specific studies, one building on another to 
increase our knowledge base. 

Any single study, no matter how good, needs to be viewed in 
the context of other research on the topic. Finding articles 
which summarize and synthesize the results of many studies, 
called literature reviews , is one good way to get a sense of the 
bigger picture that research can tell us about a given topic. 
Most research articles provide a brief review of the literature, 
and there are some specialized articles which provide compre- 
hensive literature reviews. 

When you learn about an interesting new finding, it is worth 
asking if there has been other research on this topic before, 
and if so, what the other past research has found. Some top- 
ics have had extensive research conducted on them, and 
therefore we have substantial evidence to point to. In other 
topics, there may be little research. If there haven’t been 
numerous studies, it is premature to consider that we really 
? what works. 

ERIC 



Studies in new topic areas are important, and give us an impor- 
tant indication of what direction things may be going. But they 
are certainly not definitive; we need to have numerous studies 
before researchers would say that we have a solid basis of 
evidence on what works in a given policy area. 



ADVOCATE’S CHECKLIST 



■ Has there been other past research on this same topic? 
j How much additional research? 

■ If this study’s findings are different than past research, 

[ did the researchers explain why it is different? 

■ Has there been enough high-quality research so that we 
can say we know a lot about what works in. this topic area? 
Or has there been only a little research, so we should only 
consider the research as suggestive of what might be going 
on, rather than more definitive. 



Conclusion 

Child advocates often use research to guide their policy recom- 
mendations and make persuasive arguments. To make the best 
policy choices for children, and to ensure your credibility, it is 
important to evaluate critically the research information you 
use. This fact sheet can assist you in using research effectively. 
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